Relationships of Out-of-School-Time Mathematics Lessons to 
Mathematical Literacy in Singapore and Australia 


Berinderjeet Kaur Shaljan Areepattamannil 

National Institute of Education, Singapore National Institute of Education, Singapore 
<berinderjeet.kaur@nie.edu.sg> <shaljan.a@nie.edu.sg> 

This study, drawing on data from the Programme for International Student Assessment 
(PISA) 2009, examined the relationships of out-of-school-time mathematics lessons to 
mathematical literacy in Singapore and Australia. Results of two-level hierarchical linear 
modelling (HLM) analyses revealed that out-of-school-time enrichment lessons in 
mathematics were not significantly associated with mathematical literacy in Singapore and 
Australia. Out-of-school-time remedial lessons in mathematics were negatively associated 
with mathematical literacy in Australia, while such remedial lessons in mathematics were 
not significantly related to mathematical literacy in Singapore. Learning time in out-of- 
school-time lessons in mathematics was significantly negatively linked to mathematical 
literacy in Singapore and Australia. Implications of the findings are discussed. 


Singapore took part in the Programme for International Student Assessment (PISA) for 
the first time in 2009, while Australia has been participating in PISA since 2000 
(Organization for Economic Cooperation and Development [OECD], 2010a). PISA is a 
collaborative initiative of member countries of the OECD that is aimed at assessing the 
knowledge and life skills of 15-year-old students as they approach the end of their 
compulsory period of schooling. It is a policy-driven assessment program, developed and 
directed by an international steering committee with a view to providing regular data on 
the most pressing policy issues facing educational administrators and policy makers 
worldwide (Willms, 2006). 

The sterling performance of Singaporean students on the PISA 2009 assessment and 
the Trends in International Mathematics and Science Study (TIMSS) placed Singapore on 
the list of strong performers and successful reformers in education (OECD, 2010b). 
Singapore is one of the world’s five highest-perfonning education systems (OECD, 2010b; 
Jensen, 2012). According to OECD (2010b), countries are high-performing if: 

Almost all of their students are in high school at the appropriate age, average performance is high 
and the top quarter of performers place among the countries whose top quarter are among the best 
performers in the world (with respect to their mastery of the kinds of complex knowledge and skills 
needed in advanced economies as well their ability to apply that knowledge and those skills to 
problems with which they are not familiar); student performance is only weakly related to their 
socio-economic background; and spending per pupil is not at the top of the league tables, (p. 14) 

Thus, high-performing education systems place importance on high participation, high 
quality, high equity, and high efficiency (OECD, 2010b). The average scores of 15-year- 
olds in Singapore on the PISA 2009 reading, mathematics, and science assessments attest 
to their stellar performance. Singapore ranked second in mathematics, and fourth and fifth 
in science and reading, respectively (OECD, 2010c). However, Australia ranked fifteenth 
in mathematics, and ninth and tenth in reading and science, respectively (OECD, 2010c). 
Moreover, on the TIMSS 2011 mathematics and science assessments, eighth-graders in 
Singapore ranked first and second in science and mathematics, respectively; while their 
peers in Australia ranked twelfth in both mathematics and science. Given the superior 
performance of the Singaporean students on the PISA and TIMSS assessments, it is crucial 
to examine the factors that might influence these students’ academic performance. 
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Although several factors may influence students’ academic performance (see Winne & 
Nesbit, 2010, for a review), one of the sparsely explored factors influencing student 
academic performance is out-of-school-time lessons — lessons in school subjects held 
outside normal school hours (OECD, 2011a). Out-of-school-time instruction may occur 
outside the classroom or school, and may be planned for enrichment or remedial purposes. 
As the OECD (2011b) posits, 

Secondary school students are often encouraged to take after-school classes in subjects already 
taught in school to help them improve their performance in key subjects. Students can take part in 
after-school lessons in the form of remedial “catch-up” classes or enrichment courses, with 
individual tutors or in-group lessons provided by school teachers, or other independent courses. 
These lessons can be financed publically, or can be financed by students and their families, (p. 382) 

In Singapore, 49% of the 15-year-olds attend out-of-school-time enrichment lessons in 
mathematics, while only 14% of the 15 -year-olds in Australia abend out-of-school-time 
enrichment lessons in mathematics (OECD, 2010). Further, whereas 49% of the 15-year- 
olds in Singapore participate in out-of-school-time remedial lessons in mathematics, only 
8% of the 15-year-olds in Australia take part in out-of-school-time remedial lessons in 
mathematics (OECD, 2010). However, the OECD averages for out-of-school-time 
enrichment lessons in mathematics and remedial lessons in mathematics are 17% and 18%, 
respectively. Thus, compared to Australia, large proportions of 15-year-olds in Singapore 
are taking part in out-of-school-time lessons in mathematics. Given the extent of student 
participation in out-of-school-time lessons in mathematics, it is critical to investigate the 
relationships of out-of-school-time lessons in mathematics to student performance in 
mathematics. A beher and deeper understanding of the relationship between out-of-school- 
time lessons in mathematics and student performance in mathematics may help us gauge 
whether or not investing in out-of-school-time instruction in mathematics would be 
beneficial. 

The findings of prior research on the effectiveness of out-of-school-time programs in 
mathematics are a mixed bag. For example, a study conducted by the U.S. Department of 
Education (2003) found no statistically significant effects of out-of-school-time programs 
in mathematics on elementary and middle school students’ mathematics achievement. In 
contrast, a m eta-analysis of the effectiveness of out-of-school-time programs in 
mathematics documented small but statistically significant positive effects of out-of- 
school-time programs in mathematics on student achievement in mathematics (see Lauer, 
Akiba, Wilkerson, Apthorp, Snow, & Martin-Glenn, 2006). Given the contrasting findings 
on the effectiveness of out-of-school-time programs in mathematics, it is crucial to 
examine the relationships of out-of-school-time lessons in mathematics to student 
achievement in mathematics. Hence, the purpose of the present study was to investigate the 
relationships of out-of-school-time lessons in mathematics to student performance in 
mathematics in Singapore and Australia. Specifically, the study addressed the research 
question: To what extent do out -of-school-time lessons in mathematics predict 
mathematical literacy among 15-year-olds in Singapore and Australia? 
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Method 


Data 

Data for the study were drawn from the OECD’s PISA 2009. The PISA 2009 data 
include measures of student proficiency in reading, mathematics, and science; however, 
reading was the major domain in PISA 2009, assessed with a large and comprehensive set 
of test items, whereas mathematics and science were minor domains. The Singaporean 
PISA 2009 s ample comprised of 5283 s tudents from 171 schools and the Australian 
sample comprised of 14251 students from 353 schools. 

Outcome Measure 

The outcome measure — mathematical literacy — was based on 35 t est items, and each 
response was coded as either correct or incorrect with partial credit awarded for partly 
correct or less sophisticated answers (OECD, 2012). PISA employs the mixed coefficients 
multinomial logit model (Adams, Wilson, & Wang, 1997), a multidimensional, generalized 
form of the Rasch model, to scale the PISA assessment data (OECD, 2012). 
Dichotomously scored items are scaled with the Rasch’s logistic model (Rasch, 1960), and 
items with multiple score categories are scaled with Masters’ partial credit model (Masters, 
1982). PISA also uses an imputation methodology, usually referred to as plausible values, 
to report student performance. The plausible values, an approach developed by Mislevy 
and Sheehan (1987, 1989) and based on the imputation theory of Rubin (1987), are random 
elements from the set of scores (i.e., random draws from the marginal posterior of the 
latent distribution) that could be attributed to each student (OECD, 2012). According to 
OECD (2010d), “the main reason for using plausible values is to transform discontinuous 
variables, such as test scores, into a continuous latent feature, such as underlying ability” 
(p. 151). Furthennore, because PISA uses a relatively small number of test items to 
measure student perfonnance, the use of plausible values methodology may help reduce 
biased estimates while measuring underlying ability (OECD, 2010d). We used the IEA 
International Database (IDB) Analyzer for PISA, a plug-in for SPSS, to combine the five 
plausible values and to produce their average values and correct standard errors. 


Table 1 

Descriptive Statistics 



Australia 

M 

SD 

Singapore 

M SD 

Mathematical literacy 

519.49 

90.77 

563.38 

103.42 

Gender 

.51 

.50 

.49 

.50 

Language spoken at home 

.91 

.29 

.41 

.48 

Economic, social, and cultural status (ESCS) 

.34 

.75 

-.43 

.80 

Enrichment lessons in mathematics 

.14 

.34 

.49 

.50 

Remedial lessons in mathematics 

.08 

.26 

.49 

.50 

Learning time in OST lessons in mathematics 

1.31 

.68 

2.51 

1.21 

School mean ESCS 

-.13 

1.01 

.24 

.92 
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Predictor Variables 

In PISA 2009, the student questionnaire asked students to identify the type of out-of- 
school-time lessons in mathematics they were attending, and to report how many hours 
they spent attending out-of-school-time lessons in mathematics. The out-of- school-time 
lessons in mathematics were held outside normal school hours. Moreover, such lessons 
could be held at school, at home or elsewhere, and could be taught by school or non-school 
teachers, tutors or staff. The types of out-of-school-time lessons in mathematics were: 
enrichment lessons in mathematics (1 = yes, 0 = no) and remedial lessons in mathematics 
(1 = yes, 0 = no). The learning time in out-of-school-time lessons in mathematics was rated 
on a 5 -point Likert scale ranging from 1 (I do not attend out-of-school-time lessons) to 5 (6 
or more hours a week). In addition to these predictors, control variables such as gender (0 
= male, 1 = female), language spoken at home (1 = language of assessment; 0 = another 
language), and family socio-economic status (SES) were also included in the study. The 
PISA 2009 index of economic, social, and cultural status (ESCS), an index of SES derived 
from parental occupations, parental education, and home possessions (see OECD, 2010b), 
was used as an SES measure in the current study. The descriptive statistics for all variables 
in the study are presented in Table 1 . 


Results 

Given the hierarchical structure of the PISA 2009 dataset (i.e., students nested within 
schools), hierarchical linear modelling analyses were conducted using HLM 7 for 
Windows. The random intercepts model with fixed slopes was employed (see Table 2 and 
3). Dichotomous variables were retained in their original metric. All continuous student- 
and school-level variables were centred on the grand mean. Sampling weights for students 
and schools were employed in HLM analyses to make the sample reflective of the 
population. For all analyses, the solutions were generated on the basis of full information 
maximum likelihood estimation (FIML). 

The model building followed a step-up strategy as suggested by Raudenbush and Bryk 
(2002). At the first stage, a fully unconditional model (or null or intercept-only model), 
containing only an outcome variable and no independent variables, was built. The 
intercept-only model is equivalent to a one-way random-effects analysis of variance 
(Raudenbush & Bryk, 2002). The intercept-only model was used to identify the source of 
variation within the outcome measure, mathematical literacy, by partitioning the total 
variance in the outcome measure into their within-school (level 1) and between-school 
(level 2) components. At the second stage, student-level demographic variables were added 
to the fully unconditional models to examine the statistical significance of student-level 
demographic predictors. The statistically significant student-level demographic variables 
were entered into the level 1 model along with the out-of-school-time lesson variables. At 
the final stage, the statistical significance of school-level predictor was examined by 
employing the level 2 exploratory analysis subroutine available in HLM 7. The statistically 
significant school-level variable was entered into the level 2 m odel. The proportion of 
reduction in variance as accounted for by the models served as a basis for making a 
judgment about the relative importance of student- and school-level variables (Raudenbush 
& Bryk, 2002). As recommended by Hox (2002), a variable was considered to have a small 
effect if it explained 1% variance, a medium effect if it explained 10% variance, and a 
large effect if it explained 25% variance. 
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Table 2 

Hierarchical Linear Modelling Analyses Predicting Mathematical Literacy in Singapore 



Null Model 

Model 1 

Model 2 

Model 3 


B (SE) 

B (SE) 

B (SE) 

B (SE) 

Intercept 

555.07*** 

554.77*** 

557.18*** 

553.62*** 


(5.26) 

(5.31) 

(5.17) 

(4.97) 

Student-level 





Gender 


-10.81*** 

-10.69*** 

-10.63*** 

(Female) 


(2.35) 

(2.32) 

(2.33) 

Language spoken at home 


13.35*** 

14 oi*** 

12.64*** 

(Language of assessment) 


(3.35) 

(3.35) 

(3.42) 

Economic, social, and 


25.69*** 

26.59*** 

24 59*** 

cultural status (ESCS) 


(2.11) 

(2.16) 

(2.18) 

Enrichment lessons in 



-5.55 

— 

mathematics 



(3.61) 


Remedial lessons in 



-0.18 

— 

mathematics 



(3.45) 


Learning time in OST 



-2.73* 

-3.51* 

lessons in mathematics 
School-level 



(1.32) 

(1.38) 

School mean ESCS 




87.72*** 

(14.42) 

Intercept variance (f oo) 

3764.86 

2845.40 

2764.70 

1676.11 

Level 1 variance ( 8 2 ) 

7132.68 

6713.85 

6693.43 

6696.71 

Intraclasss correlation (p) 

Variance in achievement between 

0.35 

0.30 

0.29 

0.20 

schools explained 

Variance in achievement within 

NA 

24% 

27% 

55% 

schools explained 

NA 

6% 

6% 

6% 


In Singapore and Australia, out-of-school-time enrichment lessons in mathematics 
were not statistically significantly associated with mathematical literacy. Further, out-of- 
school-time remedial lessons in mathematics were not statistically significantly related to 
mathematical literacy in Singapore, whereas such remedial lessons in mathematics were 
statistically significantly negatively associated with mathematical literacy in Australia. 
Finally, learning time in out-of-school-time lessons in mathematics was statistically 
significantly negatively associated with mathematical literacy in Singapore and Australia 
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Table 3 

Hierarchical Linear Modelling Analyses Predicting Mathematical Literacy in Australia 



Null Model 

Model 1 

Model 2 

Model 3 


B (SE) 

B (SE) 

B (SE) 

B (SE) 

Intercept 

507.01*** 

508.68*** 

515.78*** 

515.02*** 


(3.65) 

(5.71) 

(3.60) 

(3.19) 

Student-level 





Gender 


-9 73 *** 

-10.13*** 

-10.15*** 

(Female) 


( 2 . 00 ) 

(1.93) 

(1.94) 

Language spoken at home 


3.97 

— 

— 

(Language of assessment) 


(4.06) 



Economic, social, and 


30.74*** 

31.07*** 

29.12*** 

cultural status (ESCS) 


(1.75) 

(1.73) 

(1.82) 

Enrichment lessons in 



-5.42 

— 

mathematics 



(3.36) 


Remedial lessons in 



_3 1 44 *** 

-33.11*** 

mathematics 



(4.72) 

(4.70) 

Learning time in OST 



~6 96* * * 

-8.28*** 

Lessons in mathematics 
School-level 



(1.58) 

(1.41) 

School mean ESCS 




62.72*** 

( 6 . 68 ) 

Intercept variance (f oo) 

2195.74 

1543.46 

1565.18 

964.20 

Level 1 variance (<r 2 ) 

6718.34 

6329.50 

6181.27 

6186.51 

Intraclasss correlation (p) 

Variance in achievement between 

0.25 

0.20 

0.20 

0.13 

schools explained 

Variance in achievement within 

NA 

30% 

29% 

56% 

schools explained 

NA 

6 % 

8 % 

8 % 


Discussion 

The objective of the present study was to examine the relationships of out-of-school- 
time lessons in mathematics to mathematical literacy among 15 -year old students in 
Singapore and Australia. The results of the study indicated that participation in out-of- 
school-time enrichment lessons in mathematics was not related to mathematical literacy 
among 15-year olds in Singapore and Australia. Because PISA, unlike TIMSS, is not 
curriculum-driven study, the mathematics topics covered in the PISA assessment may not 
be directly linked to the school mathematics curriculum in Singapore and Australia. 

Moreover, participation in out-of-school-time remedial lessons in mathematics was not 
linked to mathematical literacy among 15-year olds in Singapore. However, participation 
in out-of-school-time remedial lessons in mathematics was negatively related to 
mathematical literacy among 15 -year olds in Australia, suggesting that students who took 
part in out-of-school-time remedial lessons in mathematics scored significantly lower in 
mathematics than did their peers who did not participate in out-of-school-time remedial 
lessons in mathematics. These results are not surprising because students who attended 
out-of-school-time remedial lessons in mathematics might be at-risk in mathematics; 
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hence, they took part in out-of-school-time remedial lessons in mathematics with a view to 
improving their performance in mathematics. Furthermore, because PISA draws its 
mathematical content from broad content areas, students who participated in out-of-school- 
time remedial lessons in mathematics may not have received training in PISA measured 
mathematics content areas at all. 

Finally, the results of the study indicated that learning time in out-of-school-time 
lessons in mathematics was significantly negatively related to 15-year olds’ mathematical 
literacy in Singapore and Australia. In other words, students who spent more time in out- 
of-school-time lessons in mathematics tended to perfonn significantly lower in 
mathematics than did their peers who spent less time in out-of-school-time lessons in 
mathematics. In conclusion, the findings of the study generally suggest that out-of-school- 
time mathematics lessons in Singapore and Australia may not be conducive for enhancing 
student achievement in mathematics. Given the growing reliance on out-of-school-time 
mathematics lessons, especially among students in Singapore, further research is warranted 
to examine the effectiveness of out-of-school-time programs and to explore its effects on 
student achievement, affect, and engagement. 
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